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Abstract 

Forbidden ordinal patterns are ordinal patterns (or 'rank blocks') that cannot ap- 
pear in the orbits generated by a map taking values on a linearly ordered space, in 
which case we say that the map has forbidden patterns. Once a map has a forbidden 
pattern of a given length Lq, it has forbidden patterns of any length L > Lq and 
their number grows superexponentially with L. Using recent results on topological 
permutation entropy, we study in this paper the existence and some basic properties 
of forbidden ordinal patterns for self maps on n-dimensional intervals. Our most ap- 
plicable conclusion is that expansive interval maps with finite topological entropy 
have necessarily forbidden patterns, although we conjecture that this is also the 
case under more general conditions. The theoretical results are nicely illustrated for 
n = 2 both using the naive counting estimator for forbidden patterns and Chao's 
estimator for the number of classes in a population. The robustness of forbidden 
ordinal patterns against observational white noise is also illustrated. 
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1 Introduction 



Ordinal patterns of length L describe the relations of 'smaller' or 'larger' 
among L consecutive points of a deterministically or randomly generated se- 
quence in a linearly ordered space. Ordinal patterns are the main ingredient 
of permutation entropy, a concept introduced both in metric and topological 
versions by Bandt, Keller and Pompe [6], that were shown to coincide with 
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their standard counterparts for piecewise monotone one- dimensional interval 
maps. Later on, the concepts of metric and topological permutation entropies 
were generalized to n-dimensional interval maps in [1] and [2], respectively, 
while preserving the main results of [6] although under different assumptions: 
ergodicity for the metric entropy and expansiveness for the topological en- 
tropy. Both generalizations parallel Kolmogorov's construction of entropy in 
dynamical systems in that they coarse-grain the state space with partitions, 
apply the corresponding definition of entropy to the resulting symbolic dy- 
namics and, lastly, take ever finer partitions. But this time the partitions used 
are product, uniform partitions (the original, one-dimensional versions even 
dispense thoroughly with partitions), making possible, albeit computationally 
demanding, the numerical estimation of metric and topological entropy. More- 
over, ordinal patterns allow a unified and conceptually simple approach both 
to metric and topological entropy, at variance with the standard approach. 

In this paper, that can be considered a second part of [2], we deal only with 
the topological permutation entropy. Having shown in [2] that this concept 
converges to topological entropy for n-dimensional expansive interval maps 
and illustrated how it can be used as estimator, we focus our attention now 
on some interesting consequences of order in dynamical systems. 

First of all, it turns out that the orbits of continuous^ n-dimensional inter- 
val maps with finite topological permutation entropy have always forbidden 
patterns, i.e., ordinal patterns that cannot occur in the orbits of the map, 
in contrast with (unconstrained) random time series, in which any ordinal 
patterns appears with probability 1. As a more practical result, it follows 
that the same happens to expansive maps with finite topological entropy. Fur- 
thermore, forbidden patterns proliferate superexponentially with length, the 
exact details being controlled by the topological permutation entropy. The ex- 
istence and growth rate of forbidden patterns was already considered in [3-5] 
but in a rather restrictive setting, namely, for piecewise monotone maps on 
one-dimensional intervals only (i.e., using the original definition of topological 
permutation entropy and results of [6]). In the present paper, we go higher 
dimensional by using the definitions and results of [2]. 

Secondly, forbidden patterns are, in general, not invariant under isomorphism 
(or conjugacy) between dynamical systems unless the isomorphism is order- 
preserving, i.e., it is an order-isomophism. This allows to further subdivide iso- 
morphic systems according to their forbidden patterns, thus opening the door 
to more restrictive definitions of equivalence among maps of n-dimensional 
intervals. 

Last but not least, forbidden patterns are robust against observational noise 
on account of being defined by inequalities. Robustness was shown in [4] to be 
instrumental for practical applications, specifically in scalar time series anal- 
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ysis. Indeed, forbidden patterns can discriminate deterministic from random 
time series when the noise is white, even if the noise level is so high that any 
trace of determinism is washed out in the return map graph. The case with 
colored noise is currently under investigation. 

This paper is organized as follows. Sect. 2 explains the basic conceptual and 
notational framework. Sect. 3 and 4 arc devoted to the study of forbidden 
ordinal patterns in higher dimensional dynamics; the former contains the the- 
oretical core and the latter deals with more practical issues, like the structure 
of forbidden patterns and their robustness against noise. Sect. 5 illustrates the 
theoretical sections with numerical evidence of forbidden patterns for Arnold's 
cat map and Henon's map, both using the naive counting estimator and Chao's 
estimator for the number of classes in a population, which is quite popular in 
mathematical biology for estimating the number of species in ecological sys- 
tems. The robustness of forbidden patterns against observational white noise 
is also addressed. 



2 Prelimirictries and previous work 

2. 1 Topological permutation entropy of information sources 

A finite-state (resp. finite-alphabet) information source with states (resp. al- 
phabet) A — {ai,a2, ...,a|A|} is a stationary stochastic process S = {Sn)n&o 
on a probability space {Q,J^,fi). Here Nq := {0} U N, Q is a non-empty set, 
JF is a sigma-algebra of subsets of f2, /i is a probability measure on the mea- 
surable space (r2,jF), and Sn '■ ^ ^ A are random variables. Such random 
processes provide models for physical information sources that must be turned 
on at some time. Observe that the possible outputs (realizations, messages,...) 
S(a;) = (sn)neNo) <^ £ i^, of the process S are points in the sequence space 



A relation < on a set X is said to be a total order (or X to be a totally ordered 
set) if < is reflexive, antisymmetric and transitive, and moreover all elements 
of X are comparable. As usual, x < y {x,y E X) means henceforth x < y and 
X ^ y. The product of linearly ordered sets, (Xi, <), (X2, <),..., (X„, <), is 
also linearly ordered via the product (also called lexicographical or dictionary) 
order: if (xi, 2:2, a;„) ^ {yi,y2, ...,yn)j then (xi, ,T2, .x„) < (t/i, ?/2- l/n) if 
(i) Xi < yi, or (ii) Xi = yi for i = l,...,k, where 1 < k < n — 1, and 
Xk+i < yk+u other conventions are of course possible. The product order 
generahzes straighforwardly to "infinite products" (i.e., sequences spaces). 



yl^" = {a = {an)nmo = («o, "i, •••) : «n e 



A}. 
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Suppose now that the alphabet A of the information soTirce S is endowed with 
a total ordering <, and let Sl denote the set of permutations on {0, 1, L—1}. 
If TT e »Sl and i— >• ttq, L — 1 t^l-Ii then we write tt = (ttq, ...,til-i)- 
Given the output (sn)neNo of S, we say that a length-L word s^^~^ — 
Sn, Sn+1, Sn+L-1 defines the ordinal {L-)pattern tt & Sl 

where, for definiteness, given Sj, Sj e A and i,j e Nq with i ^ j, 



Si -< Sj < 



^ 



or 

i < j if Si = Sj. 



For example, suppose that S is a source over the alphabet A = {1,2,3} 
ordered by size, and that we observe the output Sg = 3, 1, 1. Then, the word 
Sq defines the ordinal pattern (1,2,0). Formally one can associate with S a 
(non-stationary) random process R, = (-Rn)neNo > 

via Rn — 

|{'S'i,0 < i < n : Si < Sn}\ (|-| denotes cardinality), whose outputs ('ranks') 
are in a one-to-one relation with the ordinal patterns defined by the outputs 
of S. 

The metric and topological permutation entropies of an information source are 
defined analogously to the metric and topological entropies, but using ordinal 
patterns instead of words. In particular, if Sq~^ is shorthand for the block of 
random variables Sq, S^^i and iV(S,L) is the number of allowed ordinal 
L-patterns that S can output (or, equivalently, the number of length-L words 
of the form rg'^ — ro, ...,rL-i that can be observed in the messages of R), we 
have: 

Definition 1 The topological permutation entropy of order L of S is defined 
as 

^rop(^o^-') = ;^log7V(S,L), (2) 
and the topological permutation entropy of S as 

Kopi^) = IhnjupHl^iS^'). (3) 



The normalization factor 1/(L — 1) in (2) instead of 1/L, is due to the fact 
that single letters do not define any ordinal pattern (of course, the choice 1/L 

^ In the references, "ordinal patterns" are called "order patterns" and written be- 
tween rectangular (instead angular) parentheses. 
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leads to the same limit when L — > oo). The logarithm in (2) can be taken to 
any base > 1, the most usual bases being 2 (/i^p iii units of bits per symbol) 
and e {h^^p in units of nits per symbol). For convenience, we will use Neperian 
logarithms. 

2.2 Topological permutation entropy of maps 

For our needs it is sufficient to restrict the definition of topological permutation 
entropy of maps to interval maps. Let / be a finite interval of R'^ and / : / — ^ J 
a /^-preserving map, with being a probability measure on / endowed with 
the Borel sigma-algebra B. In order to define next the topological permutation 
entropy of /, we consider first a special coarse-graining made out of a product 
partition 

q 

k=l 

of / into N := Ni ■ ... ■ Nq subintervals of lengths Aj,k, 1 < J < ^fcj in each 
coordinate k, defining the norm = maxj^fc Aj^k of the partition o (other def- 
initions are also possible). For definiteness, the intervals are lexicographically 
ordered in each dimension, i.e., points in 7^ ^ are smaller than points in Ij+i^k 
and, for the multiple dimensions, a lexicographic order is defined, Ij k < Ij,k+i, 
so there is an order relation between all the N partition elements, and we can 
enumerate them with a single index i e {1, N}: 

t^{Ii:l<i<N}, Ii< li+i. 

Next define a collection of simple observations S' = {S^)neno with respect to 
/ with precision 

S:,{x)=z if nx)el„n = 0,1,... 

Then S'' is a stationary A^-state random process or, equivalently, an informa- 
tion source on {I,B,fi) with finite alphabet A'' = {1,...,A^}. In a dynamical 
setting, S'' is called the symbolic dynamic with respect to the coarse graining 

L. 

Definition 2 The topological permutation entropy of f is defined as 

hl,{f)= Inn K iS^). (4) 

||t||— >o 

Note that the limit (4) exists since h^^^^S'') is non-decreasing with ever finer 
partitions i. Moreover, this limit can be shown not to depend on the particular 
partition l, so that l may be taken to be uniform (i.e., a 'box partition') 
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without restriction. This being the case, we will consider in the sequel only 
box partitions. 

If / is continuous, h^^^f) is an upper bound of the topological entropy of /, 
htM [2]: 

htM < KJf). 

One of the main interests of h*^p{f) is that, under an additional hypothesis on 
/, it can be shown to coincide with the topological entropy of /, htop{f), thus 
providing eventually an estimator of it [2] . 

Indeed, let (X, d) be a compact metric space, d denoting a metric on X. A 
homeomorphism (correspondingly, a continuous map) / : X ^ X is said to 
be expansive if there exists 6 > such that d{f'"'{x), f"'{y)) < 5 for all n G Z 
(correspondingly, n G No) implies x = y. We will call 6 the expansiveness 
constant of /. Intuitively, the orbits of an expansive map / can be resolved 
to any desired precision by taking n sufficiently large. Standard examples 
of expansive maps include expanding maps on the circle, topological Markov 
chains, hyperbolic toral automorphisms and shift transformations on sequence 
spaces [7]. 

Theorem 1 [2] If I <zW^ is a compact interval and f : I ^ I an expansive 
map, then 

KopU) = htopif). 

This theorem holds also true for restrictions of expansive maps on open or 
half-open subintervals. 



3 Forbidden ordinal patterns 

Let / : X — > X be a map, where X is endowed with a hnear order <. We say 
that X G X defines the ordinal (L-)pattern tt = (7ro,7ri, ...jTTl-i) G Sl if 

r{x)<nx)<...<n~^{x) (5) 

(where /°(a;) = x and /"(x) = f{f"'~^{x))). Alternatively, we say that tt is an 
allowed {ordinal) pattern of /, or that tt is allowed, in which case 

= {x e I : x defines tt G Sl} 7^ 0- 

Furthermore, tt' G Sl is said to be a forbidden (ordinal) pattern for /, or just 
to be forbidden, if there exists no x G / defining tt', i.e., P^/ = 0. If / is 
continuous (as we will consider below), then the sets C X are open for all 
tteSl, L>2. 
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0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 



Fig. 1. The 18 allowed 4-patterns of the shift map S2 x i-^ 2a;(mod 1). For clarity, 
the allowed patterns have been written without angular parentheses nor separating 
commas. Note that the allowed patterns are 'mirrored' with respect to the central 
line, X = 1/2. 

Henceforth we consider self maps / on intervals / C M'^ endowed with lexico- 
graphical order. In this way, the ordinal patterns of / will coincide with the 
ordinal patterns of the the corresponding symbolic dynamic S*^ with respect to 
the box partition l = {Ii}i<i<N in the limit — > 0. Indeed, if x e / defines 
the pattern tt G Sl, the only way that the length-L word x, f{x), f^~^{x) 
does not define tt when observed with the precision set by the coarse-graining 
i is that at least two letters, say f"^{x) and /-'(a;), 0<i<j<L — 1, fall in 
the same box /j„ G l (since then we cannot discern the order relation between 
both letters). But all these exceptions will disappear in the limit — > 0. 

The existence of forbidden patterns is obvious in some trivial cases, e.g., when 
/ is periodic. Also, one can set up 'by hand' functions with no forbidden pat- 
terns (in particular, discontinuous one- dimensional interval maps with infinite 
many monotony segments). Let us mention in passing that the forbidden pat- 
terns of a one-dimensional maps can easily be exposed via the graphs of the 
map and its iterates. 

Example 1 Figure 1 depicts the graphs of the identity (main diagonal), the 
shift (or sawtooth) map S2 '■ x ^ 2x(modl), its second iterate, S2, and its 
third iterate, 82- The vertical, dashed lines rise at the endpoints of the intervals 
^ of points X defining the allowed patterns tt G 1S4. We conclude that S2 
has 18 allowed A-patterns and hence 6 forbidden A-patterns, namely: 

(0, 2, 3, 1) , (1, 0, 2, 3) , (1, 3, 2, 0) , (2, 0, 1, 3) , (3, 1, 0, 2) , (3, 2, 0, 1) . (6) 
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We leave as an exercise to check that S2 has no forbidden patterns of length 
3. 



We turn next to the question of finding natural conditions on / that allow the 
existence of forbidden patterns of any length. Given £ > arbitrarily small, 
there exists by Definition 2 a product partition to such that 



KJf) 



s 

< - 
2 



whenever 
such that 



Furthermore, there exists by Definition 1 a length Lq 

^Lp(S^)-^log7V(SSL)|<| 

whenever L > Lq, where iV(S', L) is the number of allowed ordinal L-patterns 
of the symbolic dynamic S' with respect to the coarse-graining i. Therefore, 
chosen i sufficiently fine and L sufficiently large, we have 



1 



Kop{f)-T^ogN{S\L) 



hence. 



N{S\L) = e 



(7) 



where the term Ol(^) depends also on L, as indicated by the subindex, and 
Cl(^) when £ — > (or — > 0). Since according to Stirling's formula. 



L! oc (L/e)^ = e^(i°s^-^)+(V2)iog 



27rL 



(where cx stands for "asymptotically"), the number of possible ordinal L- 
patterns, \Sl\ = L\, grows superexponentially with L, we conclude from (7) 
that the symbohc dynamic S' has forbidden patterns whenever hl^{f) is finite. 
Intuitively speaking, the same must happen with maps whose dynamic can be 
well approximated by simple observations. 



Theorem 2 Let I C 

Then 



be a finite interval and f : I 
hm N{S\L)^N{f,L), 



I a continuous map. 



where N{f, L) is the number of allowed patterns of f with length L. 



PROOF. Fix L > 2 and suppose that there exists an ordinal pattern 7i{x) = 
(ttq, Tri^-i) defined by x e / under / that is not visible to any symbolic 
dynamic S'' with respect to a product partition l. Since ordinal patterns are 
defined by inequalities (see (5)), there exists by continuity 77 > such that 
||x — yll < f] implies 7r{x) = TT{y), the ordinal pattern defined by y. This 
means that, in the limiting process — > 0, we do take account of all ordinal 
L-patterns that the orbits of / can produce. 
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Prom (7) it follows then, 

Corollary 3 Under the hypotheses of Theorem 2, the number of allowed L- 
patterns of f grows with L as 

|{P,^0:7re5i.}|ae^'^U/), (8) 

provided hl^{f ) < oo. 

Since checking the technical condition h*^p{f) < oo is in practice more difficult 
than checking directly the growth rate of the allowed patterns of /, we use 
now Theorem 1 to provide more natural conditions for (8). 

Corollary 4 If I (Z W is a finite interval and f : I ^ I an expansive map 
with htopif) < oo, then (8) holds true. 

The bottom line is that interval maps have forbidden patterns under quite 
general conditions and that they proliferate superexponentially with the length 
L as 

|{P^ = : TT e «Si}| oc L! - e^^*^o^^^^ 
(more on this in Sect. 4.2). 

Apart from the superexponential scaling law with L, it is quite difficult to 
make more specific statements on the forbidden patterns of a given map / 
like, for instance, the minimal length of its forbidden patterns. One important 
exception is the shift transformation on sequence spaces. 

Example 2 As in (1), let be the one-sided sequence space of the symbols 
A — {ai,a2, ...,a\A\\, where ai < a2 < ... < a\A\, being endowed with the 
lexicographic order -< defined for a' as 



ctj = for < i < n 
and ■ (9) 

Furthermore, let E : — > be the corresponding shift 

(tto, «!, a2, ...) ^ («!, a2, as, •••)• (10) 
Then one can prove [5]: 

(1) For every L < |y4| + 1, S has no forbidden patterns. 

(2) For every L > \A\ + 2, S has forbidden root patterns of length L. For 
instance if \A\ is even, then the ordinal patterns of length L = \A\-\-2 
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{0,2,...,\A\.^A\ + h...,3,l), 

(|A| + 1,|A|-1,...,1,0,2,...,|A|-2,|A|) 

and 

{\A\-1,\A\-3,...,1,0,2,...,\A\,\A\ + 1) 

are forbidden. Moreover, ifir— (ttq, tti, 7rL_2, ttl-i) is forbidden for "E, 
then its mirrored pattern 

Tl'mirrored = {^^ L-1, 7^ L-2, ■■; TTl, TTq) 

is also forbidden for E. 



4 Properties of the forbidden patterns 

To complete the picture, we briefly review in this section the three more im- 
portant properties of forbidden patterns (see also [4,5]). 

4.1 Invariance under order-isomorphims 

Since ordinal patterns are not directly related to measure-theoretical prop- 
erties, isomorphic (or conjugate) dynamical systems need not have the same 
forbidden patterns, unless the isomorphism between them preserves not only 
measure but also linear order (supposing both state spaces are linearly or- 
dered) . 

For instance, the graphical technique used in Figure 1 reveals that the logistic 
map f{x) = Ax{l — x), < x < 1, has the forbidden 3-pattern (2, 1, 0) [4], i.e., 
there are no three consecutive points in any orbit generated by the logistic 
map, forming a strictly decreasing trio. However, it follows from the general 
results stated in Example 2 that the one-sided (|, |)-Bernoulli shift [7] has 
no forbidden patterns of length 3, despite being conjugate to the logistic map 
(the interval [0, ll endowed with the measure — ). The reason is that 

the corresponding isomorphism, actually the symbolic dynamic S' : [0, 1] — »• 
{0, 1}"^° with respect to the generating partition u — {Iq — [0, \),h = [|, 1]}, 
is not order-preserving: e.g., 

s'(i) = (o,i)<s'(f) = (i), 

where the ovcrbar denotes indefinite repetition of the binary digit and binary 
strings are ordered lexicographically, while 

S^(|) = (1,1,0)>S^(1) = (1,0). 
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Definition 1 Given two linearly ordered sets (Xi, <i) and (X2, <2), two maps 
Ti : Xi ^ Xi and T2 : X2 ^ X^ and an invertible map $ : Xi — X2 such 
that $ o Ti = T2 o we say that Ti and T2 are order-is omorphic if $ is 
order-preserving (i.e., x <i y implies <2 ^{v))- 

It is trivial tliat order-isomorpliic maps liave tlie same allowed (and, hence, 
forbidden) ordinal patterns. Let us see next an interesting example of an order- 
isomorphism. 

Example 3 Let A — {0, 1, 6— 1} and consider the two-sided sequence space 
with alphabet A, 

A^ ^ {a^ {an)nGi. = (■■■, «o, «!, ■■■) : <^n e A}- 

With the notation for the 'left sequence' (a-n)„eN of the 'bisequence' a G 
and q;_|_ for its 'right sequence' (Q;n)neNo7 '"^^ define a linear order ^ in A^ 

by 

a_i_ < a'. 



or , (11) 

«_ < a'_ if = a'. 



where < between right (resp. left) sequences denotes lexicographical order in 
y^No (^j-Qgp^ ^^y), see (9). If N denotes the null set of bisequences a eventually 
terminating in an infinite string of {h — l)s in either direction, then the map 
ip : A^\J\f [0, 1) X [0, 1) = [0, 1)2 (here "\" stands for set difference) defined 
by 

(CO 00 \ 

, (12) 
71=1 n=0 / 

is one-to-one and order-preserving. As a matter of fact, the order (11) in A^ 
corresponds via ip to the lexicographical order in [0, 1)^, so we may call ^ the 
lexicographical order in A^. In sum, ip is an order-isomorphism, both A^\Af 
and [0, 1)2 being endowed with the lexicographic order. 



B{x,y) 



As way of application, consider the (non-continuous!) baker map, B : [0, 1)^ — > 
[0, 1)2, where 

\2x,\y), Q<x<\, 

^(2x-l,\y+\), \ <x<l. 

If now S denotes a two-sided shift on two-symbol sequences, then B and S are 
order-isomorphic 2 , modulo the null set M, via the 'coding map' %l) : A^\J\f — > 
[0, 1)2 given in (12). It follows that the baker map and the two-sided shift on 
two-symbol sequences have the same allowed and forbidden patterns. 



2 They are even conjugate as dynamical systems if S is the two-sided (^,|) 
Bernoulli shift and [0, 1)2 is endowed with Lebesgue measure. 
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Even more is true. First of all, one- and two-sided shifts on (bi-) sequences 
ordered lexicographically (see (9) and (11), respectively) can be proven to have 
the same forbidden patterns [5]. Furthermore, one can also prove along the 
same lines as in Example 3 that the sawtooth map x i — > 2x (mod 1) and the 
one-sided shift (10) on two-symbol sequences are order- isomorphic (modulo 
the null set of sequences terminating in 1). We conclude that the allowed and 
forbidden 4-patterns of the baker map are precisely those exhibited in Figure 
1 and listed in (6), respectively. 

4-2 Growth with length: outgrowth patterns 

According to Corollary 3, for every continuous self map / on a finite g- 
dimensional interval with finite topological permutation entropy, there exists 
TT G Sli L > 2, which cannot occur in any orbit. Moreover, if vr = (ttq, 7ri,_i) 
is forbidden for /, then it is easy to see that the 2{L + 1) patterns of length 
L + 1, 

(L, TTo, TlL-l) , (tTo, L, TTi, TTl-i) , (tTq, ...,TIl-1, L) , 

(0, TTo 1, TTl-I + 1) , (tTo 1, 0, TTi + 1, TT^-i + 1) , (tTq + 1, TTl-I + 1,0), 

are also forbidden for /. A weak form of the converse holds also true: if 
(L,7ro, ...,7rL_i), (tto,!/, ...,7rL_i), (ttq, ttlq.!, L) e Sl+i are forbidden, 
then (tto, ...,txl-i) G Sl is also forbidden. 

In turn, each of these forbidden patterns of length L + 1 belonging to the 
'first generation', will generate a 'second generation' of forbidden patterns of 
length L+2, not necessarily all different, etc.. Observe that all these forbidden 
patterns generated by tt in the nth generation have the form 

(*, TTo + n, *, TTi + n, *, *, TTi-i + n, *) e Sn (13) 

(the wildcard * stands eventually for any other entries of the pattern), with 
n = 0, 1, N—L, where N~L > 1 is the number of wildcards * G {0, 1, n— 
l,L + n,...,N -1} (with * e {L,...,N-1} if n = and * e {0, - L - 1} 
if n = N — L). Forbidden patterns of the form (13), where vr = (ttq, ttl-i) is 
forbidden, are called outgrowth forbidden patterns. If (S^*(7r) denotes the set 
of outgrowth forbidden A"-patterns of tt, the it can be proven [5] that there 
exist constants < c, d < 1 such that 

<(l-c^)Ar!. (14) 

Forbidden patterns that are not outgrowth patterns of other forbidden pat- 
terns of shorter length are called forbidden root patterns since they can be 



tt) 
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viewed as the root of the tree of forbidden patterns spanned by the outgrowth 
patterns they generate, branching taking place when going from one length (or 
generation) to the next. Thus (14) shows that alone the number of outgrowth 
A^-patterns of a given forbidden L-pattern, N > L, follows a superexponential 
growth law with N. 

4-3 Robustness against noise 



Finally, let us elaborate on the persistence of forbidden patterns when the 
observed data are distorted by small perturbations, a property we refer to as 
robustness against observational noise. As already mentioned in the Introduc- 
tion, forbidden patterns are robust against observational noise on account of 
being defined by inequalities. Were not for this property, forbidden patterns 
would not be useful in applications. 

The sort of applications we have in mind belong to the detection of deter- 
minism in imivariate and multivariate time series analysis, since random real- 
valued time series have no forbidden patterns with probability 1; see [4] for the 
intricacies of the scalar case when the sequences are finite and contaminated 
with (additive) white noise, i.e., when the time series have the form 



with / being a one-dimensional interval map and rj^. real-valued (and prop- 
erly bounded), independent, equally distributed (i.i.d.) random variables. A 
2-dimensional time series contaminated with white noise will be considered in 
the next section. The case of colored noise (i.e., random variables with corre- 
lation) is more difficult and is currently under investigation; ias a matter of 
fact, numerical sequences of the form (15) are often used to generate colored 
noise. 



5 Numerical simulations 

We demonstrate numerical evidence for the existence of forbidden ordinal 
patterns in multi-dimensional maps. Of course, direct simulation of dynamical 
systems directly yields only allowed order patterns. The failure to observe any 

given order pattern/permutation in any finite time series does not mean of 
course that it is forbidden (probability zero) but only that its probability is 
sufficiently low in the measure induced by the natural dynamics that it has 
not yet been seen. 




(15) 
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However, with sufficiently rcasonal:)lc L (as effort and memory increases rad- 
ically with L) and robust computational ability we can infer in many cases, 
robustness of forbidden patterns by examining the convergence of allowed pat- 
terns with A^, the number of order patterns (of length L) emitted by the data 
generating source. In particular, we suggest examining the logarithmic ratio of 
the cardinality of all patterns to the number of observed patterns log (L!/Pobs) 
vs logA^. If a system has a "core" of forbidden patterns, as with determinis- 
tic systems, then we expect that this ratio will decline with and eventually 
level off with increasing N, assuming the asymptotic behavior can be observed. 
Here, Pobs is the naive, biased-downward, estimator of the unknown Paiiowed- 

When is much larger than Paiiowcd) -Pobs is likely to be a good estimator, 
assuming most patterns have a reasonable probability of occurring. With in- 
creasing L, however, this is difficult to achieve practically because of memory 
limitations, as the identities and counts of each observed patterns (a subset 
of the allowed patterns) must be retained. The number of allowed patterns 
increases exponentially with L in deterministic chaos, and faster than expo- 
nentially with noise, and therefore one must increase A^, the number of iterates, 
substantially to permit a commensurately large number of distinct patterns 
to be actually observed. 

This motivates using a superior statistical estimator of Paiiowed- This equiva- 
lent problem has a significant history, motivated especially from the ecology 
community. Consider a situation where one can observe a finite sample of indi- 
vidual organisms, from a presumably large population. What is the estimated 
number of distinct species, the biodiversity, and how can we estimate this 
given the individual counts of observed species? (For reviews of approaches 
to this problem see [9,10].) This is analogous to our situation where we can 
distinguish individual order patterns but each observation is drawn from the 
natural distribution induced by typical orbits of the dynamical system. For 
our needs we wish to go reasonably deep into the undersampled regime and 
impose few probabilistic priors. We adopt the non-parametric estimator of 
Chao [11], motivated by comments in the reviews and our experience, as a 
simple but reasonably effective improvement: 

c2 

^Cha« = Pobs + 7^2, (16) 
ZC2 

where are the "meta-counts" of observations, i.e. Ci is the number of distinct 
ordinal patterns which were observed exactly once in the sample, C2 the num- 
ber which were observed exactly twice, etc. In practice this is accomplished 
by counting frequencies of observed patterns through a hash table, and in a 
second phase, counting the frequencies of such frequencies with a similar hash 
table. Note that if the sample size is particularly small (relative to what is nec- 
essary to see a substantial fraction of allowed patterns), Pchao will still be an 
underestimate. Consider that its maximum value is obtained with ci — N — 1 
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Cat map, L=10 
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Cat map L=14 
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Fig. 2. Convergence of estimated forbidden patterns with A^, cat map. Blue circles 
(o) are for Paiiowcd = -Pobs, red asterisks (*) have -Fallowed = -Pchao- Top, L = 10, 
bottom L = 14. Both figures show clear evidence of convergence to a constant, 
evidence of true forbidden patterns as A*" — > oo. In the lower figure especially, the 
improved estimator -Pchao "senses" the approach to a convergence earlier than the 
naive counting estimator. Note the differing scales on the y-axes. 

and C2 = 1, i.e. one doubleton and all remaining observations being unique 
(all unique naturally leads to an undefined estimate), and so Pchao is bounded 
by (Pobs + l)/2. Bunge and Fitzpatrick [9] call Pchao to be an "estimated 
lower bound". We believe that no statistical estimator can perform well in 
the extremely undersampled regime and there is no substitute for substantial 
computational effort when L becomes sufficiently large; however, we will see 
an improvement over the naive estimator. 

Our first numerical example is Arnold's "cat" map: (x, y) — >■ {x+y mod 1, x+ 
2y mod 1). We start with initial conditions drawn uniformly in [0, 1) x [0, 1), 
and iterate. Ordinal patterns are computed using order relations on the x- 
coordinate only. Figure 2 shows the strong numerical evidence for forbidden 
patterns characteristic of deterministic systems. As a demonstration of the 
genericity of the results. Figure 3 shows the equivalent except that the ob- 
servable upon which order patterns were computed is 3a;^ — y. Results are 
nearly identical, as one expects. 

By comparison, consider Figure 4, generated by an i.i.d. noise source (ordinal 
patterns are insensitive to changes in distribution). Here, the observed patterns 
imply convergence to zero forbidden patterns with increasing A^. More remark- 
ably the estimator Pchao senses this long before and predicts zero forbidden 
patterns with orders of magnitude lower iV, apparently because the assump- 
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Cat map, alternate observable, L=10 
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Fig. 3. Convergence of estimated forbidden patterns with A^, cat map, alterna- 
tive observable. Blue circles (o) are for Paiiowed = -fobs, red asterisks (*) have 
-fallowed = -Pchao- Top, L = 10, bottom L = 14. Both figures show clear evidence 
of convergence to a constant, evidence of true forbidden patterns as ^ oo. In 
the lower figure especially, the improved estimator Pchao "senses" the approach to 
a convergence earlier than the naive counting estimator. Note the differing scales 
on the y-axes. 



White noise L=8 
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Fig. 4. Convergence of estimated forbidden patterns with A^, i.i.d. noise. Blue circles 
(o) are for Fallowed = -fobs, red asterisks (*) have Paiiowed = -Pchao- Top, L = 8, 
bottom L = 10. Fobs shows convergence to zero forbidden pattterns; -Pchao estimates 
zero forbidden patterns well before convergence of naive estimator. 
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Henon map L=12 

17r 




16 18 20 22 24 26 28 30 

log^N 
Henon map L=19 

42 r 




16 18 20 22 24 26 28 30 

log^N 



Fig. 5. Convergence of estimated forbidden patterns with A^, Henon map. Blue 
circles (o) are for Paiiowed = -Pobs, red asterisks (*) have Fallowed = -Pchao- Top, 
L = 12, bottom L = 19. Both naive and improved estimators show convergence to 
a finite number of forbidden patterns with increasing A^. Note scale of y-axes. 



Henon map plus noise, L=10 
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Henon map plus noise L=14 
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Fig. 6. Lack of cnvergence of estimated forbidden patterns with N , Henon map with 
additive i.i.d. noise. Blue circles (o) are for Fallowed = -fobsi red asterisks (*) have 
-fallowed = -Pchao- Top, L = 12, bottom L = 19. Both naive and improved estimators 
show continued increase in allowed patterns (decrease in forbidden patterns) with 
increasing A^. 

tions made by the estimator of equiprobable patterns for both observed and 
unobserved are exactly fulfilled. 
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The cat map may be seen as too "easy" and so we turn to a chaotic system, 
the Henon map, {x,y) — (1 — ax^, bx), a = 1.4. b = 0.3, observable being the 
x-coordinate. This map is not uniformly hyperbolic, more characteristic of real 
dynamics seen in nature. In Figure 5, we see convergence to a finite core of 
forbidden patterns with larger N. Note that the performance of Pchaxa is still 
improved over the naive estimator but it is not as good as with noise, because 
with real dynamics there is a wide variation in the probability of the various 
allowed patterns, and so larger N feels the 'tail' of the distribution of rare 
patterns. By comparison consider Figure 6, which shows results from the same 
dynamics but each observable was contaminated with uniform i.i.d noise rj e 
[0,0.2). This time, increasing clearly shows increasing allowed/decreasing 
forbidden patterns, proportional to as expected with noise. The behavior 
with N cleanly distinguishes low-dimensional dynamics from noise. 

As a philosophical point it is true that the "noise" generator in a computer 
software is but a deterministic dynamical system on its own, but in practice 
it has an extremely long period and virtually no correlation, and hence if one 
wanted to see order pattern scaling different from true noise, one would need 
exceptionally long L and impractically astronomical memory requirements. 
We use a vahdated high-quahty random number generator [12] from the Boost 
C-|— I- library. 



6 Conclusion 

We showed that n-dimensional interval maps have forbidden ordinal patterns 
under the following two sufficient conditions: (i) continuity, and (ii) finite 
topological permutation entropy (Corollary 3). The second condition, that 
can be difficult to check, may be replaced by (ii') finite topological entropy if 
the first condition is replaced by (i') expansiveness (Corollary 4). In any case, 
we conjecture as a working hypothesis that the existence of forbidden patterns 
is a general feature of the interval maps encountered in practice. 

Interestingly enough, the existence of forbidden patterns can be used as an 
indicator of determinism in univariate and multivariate time series analysis, 

since sequences generated by unconstrained random processes taking values on 
intervals have no forbidden patterns with probability one. The application of 
these ideas requires some care since real sequences are finite (making possible 
that random sequences have 'false' forbidden patterns with finite probability) 
and noisy (blurring the difference between determinism and randomness) . The 
numerical simulations of Sect. 5 have provided ample evidence of all these is- 
sues, in particular of the robustness of forbidden patterns against observational 
white noise. In so doing we have also used Chao's estimator for the number of 
classes in a population. 
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